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Abstract: Conventional multiple hypothesis tests use step-up, step-down, or closed testing meth- 
ods to control the overall error rates. We will discuss marrying these methods with adaptive 
multistage sampling rules and stopping rules to perform efficient multiple hypothesis testing in 
sequential experimental designs. The result is a multistage step-down procedure that adaptively 
[tJ . tests multiple hypotheses while preserving the family- wise error rate, and extends Holm's (1979) 

step-down procedure to the sequential setting, yielding substantial savings in sample size with small 

_, loss in power. 
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Introduction 



Multiple testing problems of the sequential or, more generally, multist age natur e occur frequentl y 



in statistics. For example, in sequential fault detection and diagnosis (JNikiforov 



1995 



Lai 



2001 



1, 



after detecting that a change in the system has occurred at some time point, the task is to iso- 
late this changepoint to one of k time intervals or diagnose it as one of k change types. An- 



other area rich with examples is sequential clinical trials with multiple endpoints (e.g.. lTang et al. 



1993 



Tang and Gellerl . Il999l ) in which patients are accrued, treated, and evaluated sequentially 



with regard to k different features, none of which may have priority. Some recent new areas 
of a pplications are qua.ntitativ e finance, in em pirical tests of the profitability of trading strate- 
gies (jRomano and Woli l2005bl ) , and genomics (JGe et al.l . l2003l ) . 

Based on data coming from a parametric family Fq^O G 0, of distributions, we will be concerned 
with testing a set of hypotheses i?i, . . . , H^ C B. A hypothesis Hi is true if the true 6 lies in Hi. If 
I ^ {1, . . . , A;} is the set of indices of the true hypotheses, then the family-wise error rate (FWE) 
is defined as the probability 

P(some Hi, i G I, is rejected). (1-1) 



Othe r and somewhat less stringent notions of error rate have been proposed (cf . iHochberg and Tamhane . 
19871 ) but we focus here on FWE, the boundi ng of which is sometime s called strong error control, 



because of its prominence in clinical research (jLehmacher et al 
A s et of hypotheses iJi , . . . , Hj. is called closed if the set {Hi 



1991 



section. 



Marcus et al. 



) and other applications. 

Hk} is closed under inter- 
(|l976l ) introduced a method of testing a closed set of hypotheses Hi, ... , H^ 
that controls the FWE by requiring that there be an a-level test of every intersection hypothesis 
Di^jHi, J C {1, . . . , k}. Let H be the set of all such intersections, and note that closedness of 
{Hi, . . . ,HiS\ is equivalent to it being equal to H. Beginning with the global hypothesis n^^^Hi, 
Marcus et al.'s procedure tests the non-empty elements of H in order of decreasing dimension 
(defined as the maximum number of Hi being intersected) by their corresponding a-level tests, and 
proceeding by using the rule that H ^ H is tested if and only if all elements of H contained in H 
are tested and rejected. Multistage extension of closed tests is relatively straightforward because of 
the nature of the hypotheses and the test statistic. It is essentially a repeated closed testing pro- 



cedur e, per forming closed te s ting o f the hypotheses not yet rejected at every stage; see iTang et al. 



(11993) and 



Tang and Gelleil (|l999l ^. 



When the set of hypotheses is not closed. Holm's (|l979l ) step-down p roced ure is commonly used 
for fixed sample size problems. Beginning with a brief review of Holm's (jl979l ) procedure. Section [2] 
then proceeds to provide a multistage extension, which aims to capture the generality of Holm's 
procedure for controlling the FWE and to be able to take advantage of the closed testing structure 
when it exists. A simulation study of the proposed procedure's power and expected sample size is 



given in Section [3l and further applications and relation to the existing litertaure are discussed in 
Section HI 

2 A Multistage Step-Down Procedure 



Holml (j 19791 ) proposed the following general step-down method of testing Hi, ... ^H^ that does 
not assume closedness. Although Holm's procedure has been criticized for lack of power in some 
settings, it does preserve the FWE without making any assumptions about the structure of the 
hypotheses or correlations between the individual test statistics, only that for each hypothesis Hi 
there is a computable p-value pi such that 

P{Pi < a\Hi) < a (2.1) 

for all < a < 1. The a-level Holm's procedure proceeds as follows. Compute and order the 
p-values ^i(i) < . . . < ^i(fc). For j = 1, . . . , A:, if 

%)>a/(A;-i + l), (2.2) 

then accept -ffj(j), • • • , H^i^y, otherwise, reject -f^i(j) and move on to stage ? + 1 (if j < k). A simpl e 



proof that the FWE of Holm's procedure is bounded by a is given in lLehmann and Romano! (|2005l ). 

As noted above, when closedness exists it is unnecessary to use the step-down correction (j2.2p . 
or any Bonferroni-type correction for that matter. However, it is illuminating to now consider how 
the step-down procedure is related to closed testing procedures. When closedness exists, Marcus 



et al.'s (jl976l ) procedure can be viewed as a special case of Holm's step-down procedure in the 



following sense. Assume that the p-values "respect" the closedness in the sense that 

HiQHj => pi<pj. (2.3) 

In this case Holm's procedure will test the elements oi H = {Hi, . . . ,Hk} in order of decreasing 
dimension (provided we agree to use dimension to break any "ties"). Assuming that closedness 
and (|2.3|) hold, we make two slight modifications of Holm's procedure to utilize these properties. 
First, upon rejection of Hiij\, we accept any of the remaining hypotheses whose complements 
are implied by -ffiQ), i.e., accept any H^^j j^i-^, . . . , -ffj(/;) containing H^i-., the complement of -ffj(j)- 
This does not change the FWE < a bound since (j2.3p guarantees that the intersection of all true 
hypotheses, denoted by G, is the first true hypothesis tested. Otherwise, if G has already been 
accepted, then all true hypotheses are subsequently accepted according to this rule. Next, note 
that the Bonferroni-type correction in (j2.2p is now unnecessary, i.e., the right-hand-side of (j2.2p 
may be replaced by a while maintaining FWE < a, since, letting jq denote the rank of the p-value 



associated with G, 



FWE = P(G rejected) 
< a. 



We now introduce a multistage generalization of Holm's (jl979l ) step-down procedure. As 
with the original version of Holm's procedure, we make no assumptions about the structure of 
the set of hypotheses Hi, ... , H^ to be tested, and the only assumptions are about the family of 
av ailable tests of the i ndividu al hypotheses through their significance levels, following the approach 
of iRomano and Wola (|2005al ). That this procedure satisfies FWE < q is proved in Theorem 1. 



For each hypothesis Hi, assume there is a sequential test statistic Tj_„, a (non-random) critical 
value function Cn{p) that is non-increasing in /> € (0, 1), and a set N of possible sample sizes such 
that 

Cn{p)] >^<P (2.4) 



sup Pe sup [Ti^n 

eeHi \neN J 

for all < p < 1. This requirement is the multistage analog of (j2.ip . The existence of such a family 

of sequential test statistics may at first seem restrictive, but in many settings there are natural 

choices for the Tj_„; see the examples in Sections [3] and HI The use of the set N of possible sample 

sizes allows for the possibilities of fully sequential (e.g., N = {1, 2, 3, . . .}) or group sequential (e.g., 

N = {m, 2m, 3?7i, 4m, 5m} for some m) in both the truncated and non-truncated settings. 

The a- level multistage step-down procedure with no more than k stages is defined as follows. 
Let I\ = {1, . . . , /c}, no = 0, and let | • | denote set cardinality. For j = 1, . . . , /c: 



1. Sample up to 



Uj = inf In ^ N : n> rij-i and maxTj „ > C„(a/|/,|) > . (2-5) 



2. Order the test statistics 



Ti{j,l),n, > Ti{j,2),n, >■■■> Ti(^j,\I^\),n,- 



3. Reject -ffj^,!), • • • , Hi^j,mj), where 

max < m, > 1 : min 



rrii 



l<i<m 



Ti[jl),nj - Crij 



a 



\I,\-i + l 



> 



(2.6) 



4. Stop if j = k, if Uj = supA^, or if all remaining hypotheses contain the complement of 
some rejected hypothesis. Otherwise, let Ij+i be the indices of the remaining hypotheses and 
continue on to stage j + 1. 



Theorem 1. The multistage step-down procedure satisfies FWE < a. 

Proof. Let / be the indices of the true hypotheses. If an error occurs, then for some Ij 3 / and 
index i such that |/j| — ^ + 1 > |/|, 



max sup [Ti^n - C'„(a/|I|)] > 0. 



which imphes that 



Then, using the Bonferroni inequahty and ([27 

FWE < P (max sup [Ti,„ - C„(a/|/[)] > ) 

< y^p(snp[T,^n-Cnia/\I\)]>o) 

1^1 VneTV ; 

< j;a/|/|=a. 

D 

We point out additionally that the procedure may stopped at Step 4 at any point as long as 
the remaining hypotheses are accepted, since this action can only serve to decrease the FWE. This 
feature may be of use in clinical trial applications; see the last paragraph of Section [4. 2i 

Theorem 1 holds regardless of the structure of the Hi or the joint distribution of the test 
statistics Tj^„. When the Hi are closed, the above multistage step-down procedure can be modified 
slightly to take advantage of this additional structure, analogous to the discussion in the second 
paragraph of this section for the fixed-sample case. To this end, assume that the set /fi, . . . , Hj^ is 
closed, and that the test statistics respect the closedness in the sense that 

Hi C Hj =^ Ti^n > Tj^n for all n G TV. (2.7) 

As in the fixed-sample case, this guarantees that the hypotheses are analyzed by decreasing dimen- 
sion. First, we modify the multistage step-down procedure by changing Step 3 to: 

3.' For f. = 1, . . . , ?7ij : Reject i7j(j £)^^ , and accept any remaining hypotheses containing H^,- ^n „ . • 

Next, we replace the fractions of a in (|2.5|) - (|2.6p by a. These modifications do not cause violation 
of FWE < a by the same proof given in the second paragraph of this section, proving the following. 

Theorem 2. If the set of hypotheses Hi^ . . . , H^ is closed and the test statistics Ti^n satisfy (j2.7p . 
then the multistage step-down procedure with Step 3 replaced by 3' and a as the argument of Cn 
in (123D-(I2SD satisfies FWE < a. 



3 Power and Expected Sample Size 



The Holm step-down procedure's attractive quality is its generality, i.e., no assumptions about 
the structure of the hypotheses Hi,. . . ,Hk or the individual test statistics, other than (j2.ip . are 
necessary. This generality is provided by the Bonferroni-type adjustment (j2.2p which also can cause 
Holm's procedure to be conservative, in terms of FWE and power, relative to procedures that take 
into account correlations between the individual test statistics. This conservativeness is shared 
by the multistage step-down procedure because of its use of an analogous step-down rule (j2.6p . 
However, as pointed out above, the utility of either the multistage or fixed sample step-down 
procedure lies in cases where such correlations are difficult to model. 



Table 1: A 3-Endpoint Trial 



Procedure 


(^i,/^2,;>) 


EM 


P(rej. Hi) 


P(rej. H,) 


P(rej. H,) 


FWE 


H 




105 


1.7% 


1.7% 


0.8% 


4.0% 


Mult 


(0,0, .5) 


104.7 


1.6% 


1.6% 


1.8% 


4.9% 


MultH 




104.6 


1.5% 


1.5% 


2.0% 


4.8% 


H 




105 


2.3% 


2.3% 


76.0% 


4.4% 


Mult 


(0,0, .75) 


98.4 


1.7% 


1.7% 


79.9% 


3.2% 


MultH 




98.3 


2.1% 


2.1% 


80.7% 


4.2% 


H 




105 


2.5% 


95.7% 


2.1% 


4.4% 


Mult 


(0,.65,.5) 


96.9 


1.6% 


94.5% 


1.9% 


3.4% 


MultH 




96.9 


2.2% 


94.6% 


2.9% 


4.9% 


H 




105 


4.3% 


82.9% 


83.9% 


4.3% 


Mult 


(0,.5,.75) 


92.8 


1.5% 


76.3% 


80.3% 


1.5% 


MultH 




92.3 


3.2% 


79.9% 


85.2% 


3.2% 


H 




105 


83.4% 


83.4% 


3.8% 


3.8% 


Mult 


(.5, .5, .5) 


93.5 


76.3% 


76.3% 


1.8% 


1.8% 


MultH 




93.1 


79.6% 


79.6% 


2.7% 


2.7% 


H 




105 


70.9% 


70.9% 


86.8% 


NA 


Mult 


(.4, .4, .75) 


89.9 


55.3% 


55.3% 


80.2% 


NA 


MultH 




89.3 


64.7% 


64.7% 


85.4% 


NA 


H 




105 


88.6% 


88.6% 


90.0% 


NA 


Mult 


(.5, .5, .75) 


87.2 


76.4% 


76.4% 


80.0% 


NA 


MultH 




86.1 


84.4% 


84.4% 


87.0% 


NA 



Consider a multistage step-down procedure with maximum sample size n = max N. Relative to 
the fixed-sample Holm step-down procedure of size n, the multistage procedure will have a reduction 
in expected sample size, provided the set A^ is chosen reasonably. But, by the Neyman-Pearson 
lemma, the power of the multistage procedure for rejecting a given false hypothesis cannot exceed 
the power of the fixed-sample Holm procedure. However, as shown by the following simulation 
study, this loss in power is usually slight while the savings in expected sample size tends to be 
substantial. 

Consider a three-endpoint clinical trial where two of the endpoints concern continuous data 
and the third concerns probability of a certain binary outcome. For example, let the data be 
Xi = {Xii,Xi2,Xi^), i = 1,2,..., where for j = 1,2, the Xji are i.i.d. normal random variables 
with unknown mean fij and variance 1, and the X^i are independent Bernoulli random variables 
where p = PiX^i = 1) is unknown. Suppose the Xi represent clinical treatment outcomes for three 
endpoints of interest, and it is desired to test efficacy of the treatment in the form of the three 
one-sided null hypotheses 

Hi:fii<0, H2: fi2<0, H^:p> 1/2. 

In cases such as this, the correlation between the components of Xi is likely to be unknown or 
difficult to model. In the following simulation study we compare the performance of the mul- 
tistage step-down procedure with two other procedures in the three cases of independent, posi- 
tively correlated, and negatively correlated components of Xi. Table 1 contains the results for 
the independent case; the other two are discussed below. Whatever the correlation between the 
components, the three procedures evaluated are equally applicable since they do not depend on 
the correlation structure of the individual hypotheses. For Holm's step-down procedure, we use 
standard a = .05-level likelihood ratio tests of Hi, H2, H3 of size n = 35 to have power around 90% 
at {fii,iJ:2,p) = (.5, .5, .75). For the multistage step-down procedure, we use the same test statis- 
tics in one-sided group sequential tests with A^ = {26, 29, 35} and use a normal approximation for 
Y2iX3i to compute Cn{p) to satisfy (j2.4p . Table 1 contains the expected sample size, probability 
of rejecting each Hi, and FWE (when uf^^Hi is true) for the Holm procedure (denoted by H) 
and the multistage step-down procedure (denoted by MultH). To see the effects of the step-down 
rule (j2.5p - (|2.6p . we also include the multistage test (denoted by Mult) identical to MultH but with 
a divided by A; = 3 in place of the larger fractions of a in (J2.5p - (j2.6p : see the first paragraph of 
the Section 14. li Each entry in Table 1 is computed from 50,000 simulation runs. The last six 
FWE entries are marked NA (not applicable) because none of the null hypotheses are true for 
those parameter values. The multistage procedures Mult and MultH show substantial savings in 
expected size sample over the Holm procedure's fixed sample size of 105 at significant deviations 
from the "worst-case" null (/xi, ^2,p) = (0, 0, .5). The expected sample size of Mult and MultH are 
nearly identical, the former being somewhat larger due to its larger critical values in ()2.5p - ()2.6p . 

6 



For the same reason MultH has shghtly higher power than Mult; in particular, see the last six rows 
of Table 1. Although the power of MultH is lower than H due to its multiple looks, this difference is 
slight, usually within a few percentage points. This relative relationship does not change when the 
components of Xi are correlated, it simply tends to decrease slightly in magnitude when positively 
correlated, and increase when negatively correlated. For example, when {fii,fj,2,p) = (0, .5, .75) and 
the two normal components have a correlation coefficient of .75, P(reject Hi) increases to 4.5%, 
2.0%, and 3.6% for H, Mult, and MultH, respectively, while the power P(reject H2) decreases to 
80.7%, 73.4%, and 77.0%, respectively. Here Mult and MultH have expected sample size of 94.7 
and 94.2. 

4 Applications and Discussion 



4.1 Sequential /c- Hypothesis Testing 

A straightforward application of ()2.4p to sequential testing of k null hypotheses is to use Bonferroni's 
inequality so that the k hypotheses can be treated separately by setting p = a/k in (12.41) . This 
approach to sequential multiple testing has been taken by many authors. iPaulsonI (|1964| ) noticed 
that further sample size savings might be possible by eliminating (rejecting) some hypotheses during 
the course of the experiment, similar to the test Mult in the preceding section. We have refined the 
multistage rejection of hypotheses by using Holm's step-down procedure to sharpen the Bonferroni 
bounds. In fact, when the set of hypotheses is closed, a slight modification of our multistage test 
can dispense with Bonferroni bounds, as shown in Theorem 2. 



Sequential multiple hypothesis testing dated back to lSobel and Waldl (j 19491 ) in deciding which 
of three simple hypotheses Hi : 6 = —d, H2 : 9 = 0, or H3 : 9 = d is true about a normal 
mean 9, for a fixed value d > 0. This is basically a classification problem. The Sobel-Wald test 
combines two sequential probability ratio tests (SPR Ts) for diff e rent p airs of the three hypotheses, 
the comparison of Hi versus ^^3 being superfluous. lArmitagd (|l95d ) generalized the Sobel-Wald 
problem to k hypotheses, corresponding to an error matrix aij = Pi{i = j) for i ^ j, where Pi 
denotes the probability measure under Hi and i is the hypothesis chosen by the test. Armitage's test 
combines the corresponding (2) SPRTs for the k hypotheses, which leads to stopping boundaries 
with slope ±1 /2 in the (n, 5'n /rf)-plane, where Sn = Yl'i=i -^i i^ ^^^ ^^^^ °f ^^^ ^^^^ '^ i.i.d. normal 



observations. ISimonsI (|l967l ) considered a generalization of a special case of Armitage's test for 
A; = 3 in which the stopping boundaries' slopes can be chosen arbitrarily. Whereas the Sobel- 
Wald, Armitage, and Simons tests stop and decide on a hypothesis Hj when all the component 
SPRTs simultaneously prefer Hj to all other alternatives, iLordenl (jl972l . 119761 ) introduced various 
multistage tests that decide on Hj only when all other hypotheses can be rejected, based on 



genera li zed likelihood ra tios. Lorden's work was extended to composite hypotheses by iPavlov 
(|l988l ). lEisenbera (|l99ll ) gives a detailed summary of these problems. 



PaulsonI (jl963l ) introduced another generalization of the Sobel-Wald test, considering k > 2 
intervals (—00,^1), {61,62), ■■ ., {6k-i,oo) and testing sequentially to which interval 6 belongs. In 
its symmetric form, Paulson's test stops the first time the interval {un,Vn) is contained in one of 
the intervals (—00, 61 + 6), (6^-1 — 6, 00), or {6i — 6, ^j+i + 5), i = 1, . . . ,k — 2, where (5 > is a 
chosen parameter and 



Un = max Um = max [Sm/'m — 5/2 — A/m) 

l<rra<n l<m<n 

Vn = min Vm = rnin (Sm/fn + 5/2 + A/m) 

l<m<n l<m<n 



(4.1) 
(4.2) 



in which A > is a critical value that can be chosen to give desired coverage probability. Paul- 
son's procedure can be viewed as a special case of the multistage step-down procedure, as follows. 



Ai) . 



r(i) . 



{i) 



Defining H^'' : 6 = 6^ - S and Hi'' : 6 = 6i {1 < i < k - 1), the one-sided SPRT of H^' versus H^ 

(i) 

stops sampling and rejects Hq if 



r(0 



n[{6i - 6) + 6,]/2 > A, 



(4.3) 



for some critical value A > 0. Dividing both sides of 



by n and rearranging terms gives 



A/n < Sn/n - [Oi - 5/2) = Sn/n - 5/2 - {6i - 5), 



(i) 



which by (|4.ip is equivalent to Un > 6i — 5. Similarly, the one-sided SPRT oi Hq : 6 = 6 + 5 

(i) ~ (i) ~ 

versus H^ stops sampling and rejects Hq if ti^ < 0j + 5. Hence running the multistage step-down 
procedure until a coherent classification is made is precisely Paulson's procedure. In the small- 
probability event that no coherent classification is made - say, if at some stage a hypothesis is 
rejected containing the complement of a previously r ejected hypothesis - classification based on 
Sn/n or randomization can be used, as iPaulsonI (jl963l ) suggests. 



Although we have used Hq and Hq to denote null hypotheses for combining SPRTs above, 
there is no natural notion of a "null" hypothesis in the actual classification problem. We could have 
populated our list of hypotheses to test in any way that suited our needs. Thus it is also perhaps 
interesting to consider what closed testing has to say about the Sobel-Wald-Paulson problem. In 
particular, for the case fc = 3 in the Paulson problem, let Hi : 6 < 61, H2 : 6 > 61, H-^ : 6 < 62, 
and H4 : 6 > 62- The set {Hi,H2,H^,Hi} is not closed, but adding H5 = H2 D H3 = {61,62) 
to the list of hypotheses completes its closure H = {Hi, H2, H3, H4, H^}. Since all intersections 
of dimension 3 or higher are empty, the closed testing principle suggests beginning by testing the 
hypotheses of dimension 2, namely Hi = Hi Ci H^ = {—00,61), H4 = H2 n H/i = {62,00), and 
H5 = {61,62), which is of course the original Paulson problem with k = 3. Hence it seems that the 
closed testing principle does not give any new insight into the Paulson problem. 



A closely related problem to s equential fe-hypo thesis testing is selecting the one of k normal 
populations with the largest mean. iBechhoferl (jl954l ) considered this problem when the variance of 
the observations is known, and proposed a fixed samp le procedure that comp ares the sample means 
of the individua l populations. F or un known variance 
procedure, and 



Bechhofer et al. 



Robbins et al. 



(jl954l ) proposed a two-stage 



(|1968l ) proposed a sequential procedure with improved efficiency, 
also based on sample means. The speci al case of when t he mean i s an i nteger was considered by 
RobbinsI (1970|), and later generalized by iMcCabd ( 1973 ). iRobbind (jl970l ) introduced the notion of 



Khan 



19731) studied the asymptotic efficiency 



"distinguishability" of a family of populations, and 
of stopping rules that distinguish within such families. iMukhopadhvavl (jl983l ) proposed likelihood- 
based methods for the largest mean problem, and used Khan's results to show asymptotic efficiency. 
Lik elihood-based methods were shown t o be useful in a number of related selection pr oblems as well 



see 



( 200 



Mukhopadhvav and Solankvl (j 19941 ). Further references are given in Section 2 of 
1). 



Chan and Lai 



4.2 Multiple Endpoint Clinical Trials 

The multistage step-down procedure provides a general method of testing multiple endpoints in 
clinical trials. The adaptive rejection times ()2.5p and rejection rule ()2.6p have the effect of adaptively 
"dropping" (i.e., rejecting) hypotheses when enough information has accumulated to do so, to focus 
on the statistically most interesting endpoints. As discussed above, when closedness exists, closed 
testing methods should be used since they are a re in general more power ful th an step-down method s 
(simulations verifying this were conducted by iLehmacher et al.l (jl99ll ) and iTang and LinI (j 19971 )) 



because they forgo the need for Bonferroni-type corrections, such as in (|2.2p . ()2.5p . and (j2.6p . 
The multistage step-down method could be useful in cases where clo sedness does not e xist. For 
example, in clinical trials for AIDS treatments, it is common (e.g., iFischl et al.l . 119871 ) to have 



multiple endpoints of both the continuous and categorical types, like CD4 (T-cell) level, which 
is commonly modeled as a normal random variable, and the binary indicator of opportunistic 
infectious disease like a cold, modeled as a Bernoulli random variable. Moreover, if one or some 
subset of the endpoints is of primary interest, the multistage step-down procedure can be used as 
a pilot or screening phase with the option of immediately stopping and proceeding to secondary 
testing when one of the primary hypotheses is rejected. That this does not increase the FWE is 
pointed out following the proof of Theorem 1. The multistage step-down procedure provides a 
general framework that can be applied to multiple testing in these clinical trials. 
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